Augmenting Reality For A User

ABSTRACT

Augmenting reality for a user including creating a model of a region of the user&#39;s environment in a direction of the user&#39;s field of vision; determining, in dependence upon the model of the region of the user&#39;s environment, that an object located within the user&#39;s environment is invisible to the user; determining, in dependence upon the direction of the user&#39;s field of vision, the object&#39;s location, and the model of the region of the user&#39;s environment, a location to display a representation of the object on a transparent display in front of the user&#39;s field of vision; and displaying the representation of the object at the determined location on the transparent display in front of the user&#39;s field of vision.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention is data processing, or, more specifically,methods, apparatus, and products for augmenting reality for a user.

2. Description of Related Art

The development of the EDVAC computer system of 1948 is often cited asthe beginning of the computer era. Since that time, computer systemshave evolved into extremely complicated devices. Today's computers aremuch more sophisticated than early systems such as the EDVAC. Computersystems typically include a combination of hardware and softwarecomponents, application programs, operating systems, processors, buses,memory, input/output devices, and so on. As advances in semiconductorprocessing and computer architecture push the performance of thecomputer higher and higher, more sophisticated computer software hasevolved to take advantage of the higher performance of the hardware,resulting in computer systems today that are much more powerful thanjust a few years ago.

Computer systems today may be capable of gathering information thatexceeds the information gathering capabilities of a user. In some casesit may be beneficial to use this information gathered by computersystems to supplement the user's current knowledge at any given time.One such case in which information gathered by computer systems may beuseful to supplement the user's current knowledge is when user isincapable of viewing objects within the user's field of vision. Examplesof objects in which a user may be incapable of viewing may include aball behind a car, an item in a closed cabinet, a person behind a door,and so on. If computer systems have gathered information about theobjects not viewable by the user it would be useful to supplement theuser's knowledge with the gathered information.

SUMMARY OF THE INVENTION

Methods, apparatus, and products for augmenting reality for a user aredisclosed that include creating a model of a region of the user'senvironment in a direction of the user's field of vision; determining,in dependence upon the model of the region of the user's environment,that an object located within the user's environment is invisible to theuser; determining, in dependence upon the direction of the user's fieldof vision, the object's location, and the model of the region of theuser's environment, a location to display a representation of the objecton a transparent display in front of the user's field of vision; anddisplaying the representation of the object at the determined locationon the transparent display in front of the user's field of vision.

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescriptions of exemplary embodiments of the invention as illustrated inthe accompanying drawings wherein like reference numbers generallyrepresent like parts of exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 sets forth a functional block diagram of a system for augmentingreality for a user according to embodiments of the present invention.

FIG. 2 sets forth a block diagram of automated computing machinerycomprising an exemplary pair of augmented reality glasses useful inaugmenting reality for a user according to embodiments of the presentinvention.

FIG. 3 sets forth a flow chart illustrating an exemplary method foraugmenting reality for a user according to embodiments of the presentinvention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary methods, apparatus, and products for augmenting reality for auser in accordance with the present invention are described withreference to the accompanying drawings, beginning with FIG. 1. FIG. 1sets forth a functional block diagram of a system for augmenting realityfor a user (140) according to embodiments of the present invention. Thesystem of FIG. 1 operates generally for augmenting reality for a useraccording to embodiments of the present invention

The system of FIG. 1 includes an augmented reality engine (122)installed on a server (106). An augmented reality engine is a module ofcomputer program instructions useful in augmenting reality for a useraccording to embodiments of the present invention. The augmented realityengine (122) operates for augmenting reality for a user (140) bycreating a model (124) of a region of the user's environment in adirection (133) of the user's field of vision (132); determining, independence upon the model (124) of the region of user's environment,that an object (136) located within the user's environment is invisibleto the user (140); and determining, in dependence upon the direction(133) of the user's field of vision (132), the object's location (128),and the model of the region of the user's environment (124), a locationto display a representation of the object on a transparent display (102)in front of the user's field of vision (132).

A user's field of vision is an area which a user is capable of viewingwith no obstruction through the transparent display (102). A user'sfield of vision may vary depending upon the implementation of thetransparent display. In this specification the exemplary transparentdisplay is implemented in a pair of glasses for clarity only. Atransparent display may also be implemented in goggles, binoculars, awindshield in an automobile, and so on as will occur to those of skillin the art. The magnification and angular visibility, and therefore theuser's field of vision through the transparent display, may vary betweenimplementations of the transparent display.

A region of the user's environment is an area having dimensions that maybe defined by a user and is typically directed in the same direction asthe user's field of vision. A direction of the user's field of vision isthe orientation of the user. If a user is oriented north, for example,the direction of user's field of vision is north. A region may bedefined by a user, for example, as a the area within a 170 degreecircular arc protruding from the user having a radius of one mile with abisection of the arc aligned with the orientation of the user. Typicallythe region of the user's environment in the direction of the user'sfield of vision is an area that closely resembles the user's actualfield of vision.

The region of the user's environment in the direction of the user'sfield of vision may include objects currently invisible to the user aswell as objects currently visible. An object (136) invisible to the user(140) is an object within the region of the user's environment in thedirection of the user's field of vision that is not currently visible tothe user. Exemplary reasons that an object may be invisible to a userinclude that an obstruction (134) exists between the user (140) and theobject (136), the object is in an area of low light, the weather impedesthe user's vision of the object, and other reasons as will occur tothose of skill in the art.

A model of the region of the user's environment in the direction of theuser's field of vision may be a three dimensional map of the user'senvironment defined by a grid representing latitude, longitude, andaltitude. The augmented reality engine (122) may create the model (124)of the region of the user's environment in dependence upon images of theuser's environment and global positioning system (‘GPS’) coordinates.The augmented reality engine of FIG. 1, for example, may create themodel (124) of the region of the user's environment in dependence uponthe images (126) and the GPS coordinates (142) received from a satellite(116). The satellite (116) of FIG. 1 transmits such images (126) andcoordinates to the server (106) through a wireless communicationchannel. Readers of skill in the art will recognize that any number ofdevices capable of capturing images may transmit such images (126) tothe augmented reality engine for creating the model of the region of theuser's environment. Readers of skill in the art will recognize thatthere are other ways of creating a model of the region of the user'senvironment including, for example, using relief map data, blueprintdata, or other data describing the region of the user's environment inthe direction of user's field of vision.

In the system of FIG. 1 the augmented reality engine (122) receives thelocation (128) of the object (136) from the unmanned aerial vehicle(‘UAV’) (130) through a wireless communications channel. Readers ofskill in the art will recognize that the UAV is shown here for clarity,but the augmented reality engine (122) may receive the location (128) ofthe object (136) in various ways. The augmented reality engine (122),for example, may be pre-configured with the location of a stationaryobject, may receive the location from a GPS tracking device installed onthe object, or in other ways as will occur to those of skill in the art.

The system of FIG. 1 also includes a pair of augmented reality (‘AR’)glasses that includes the transparent display (102). The AR glasses(104) include a display module (not shown in FIG. 1). The display moduleis a module of computer program instructions that operates foraugmenting the reality of a user by displaying the representation (138)of the object (136) at the determined location on the transparentdisplay (102) in front of the user's field of vision. A representationof an object may be implemented in various ways. A representation of theobject may, for example, include an image of the object or a symbolrepresenting the object. A representation of the object may be an opaqueimage or a translucent image. A representation may be implemented in anyshape or size.

The arrangement of servers and other devices making up the exemplarysystem illustrated in FIG. 1 are for explanation, not for limitation.Data processing systems useful according to various embodiments of thepresent invention may include additional servers, routers, otherdevices, and peer-to-peer architectures, not shown in FIG. 1, as willoccur to those of skill in the art. Networks in such data processingsystems may support many data communications protocols, including forexample TCP (Transmission Control Protocol), IP (Internet Protocol),HTTP (HyperText Transfer Protocol), WAP (Wireless Access Protocol), HDTP(Handheld Device Transport Protocol), and others as will occur to thoseof skill in the art. Various embodiments of the present invention may beimplemented on a variety of hardware platforms in addition to thoseillustrated in FIG. 1.

Augmenting reality for a user in accordance with the present inventionis generally implemented with computers, that is, with automatedcomputing machinery. In the system of FIG. 1, for example, the serverand AR glasses are implemented to some extent at least as computers. Forfurther explanation, therefore, FIG. 2 sets forth a block diagram ofautomated computing machinery comprising an exemplary pair of augmentedreality glasses (104) useful in augmenting reality for a user accordingto embodiments of the present invention. The pair of augmented realityglasses (104) of FIG. 2 includes at least one computer processor (156)or ‘CPU’ as well as random access memory (168) (‘RAM’) which isconnected through an expansion bus (160) to processor (156) and to othercomponents of the computer.

Stored in RAM (168) is a display module (120), computer programinstructions for displaying representations of an object on thetransparent display (102). The display module (120) is capable ofrendering a display of the representation of the object by using aprojector (180). The projector (180) renders a display on thetransparent display (102) for viewing by the user. The exemplaryprojector (180) of FIG. 2 is implemented with Digital Light Processing(‘DLP’) technology originally developed at Texas Instruments™. Theprojector in FIG. 2 is connected to other components of the exemplarypair of AR glasses (104) through a DLP adapter (209) and a Video Bus(164). Other technologies useful in implementing the projector (180) mayinclude liquid crystal display (‘LCD’) technology and liquid crystal onsilicon (‘LCOS’) technology.

Also stored in RAM (168) is an operating system (154). Operating systemsuseful in augmented reality devices according to embodiments of thepresent invention include UNIX™, Linux™, Microsoft XP™, MicrosoftVista™, AIX™, IBM's i5/OS™, and others as will occur to those of skillin the art. Operating system (154) and display module (120) in theexample of FIG. 2 are shown in RAM (168), but many components of suchsoftware typically are stored in non-volatile memory also.

The exemplary pair of AR glasses (104) of FIG. 2 includes a globalposition system (‘GPS’) receiver (108). A GPS receiver (108) is devicethat receives from a GPS satellite network a steady stream of GPS datadescribing the receiver's location. The exemplary GPS receiver (108) ofFIG. 2 may transmit a user's location to an augmented reality enginerunning on the server (106) for creating a model of the region of theuser's environment and for determining a location to display therepresentation of the object.

The exemplary pair of AR glasses (104) of FIG. 2 also includes a digitalgyroscope (110). A gyroscope is a device for measuring or maintainingorientation based on the principle of conservation of angular momentum.Digital gyroscopes may be used to determine the orientation of a devicein which the gyroscope is included. In the example of FIG. 2 the digitalgyroscope may be used to determine a user's orientation. That is, thedigital gyroscope of FIG. 2 may be used to determine the direction ofthe user's field of vision. That direction along with the location ofthe user from the GPS device may be used to define the region of theuser's environment in the direction of user's field vision.

The exemplary pair of AR glasses (104) of FIG. 2 includes one or moreinput/output adapters (178). Input/output interface adapters incomputers implement user-oriented input/output through, for example,software drivers and computer hardware for controlling output tospeakers (171), as well as user input from user input devices such as amicrophone (176).

The exemplary pair of AR glasses (104) of FIG. 2 includes acommunications adapter (167) for implementing data communications (184)with other devices including, for example, a personal digital assistant(‘PDA’) (112), a server (106), or a laptop (116). Such datacommunications may be carried out serially through RS-232 connections,through external buses such as USB, through data communications networkssuch as IP networks, and in other ways as will occur to those of skillin the art. Communications adapters implement the hardware level of datacommunications through which one computer sends data communications toanother computer, directly or through a network. Examples ofcommunications adapters useful for augmenting reality for a useraccording to embodiments of the present invention include modems forwired dial-up communications, Ethernet (IEEE 802.3) adapters for wirednetwork communications, and 802.11g adapters for wireless networkcommunications.

For further explanation, FIG. 3 sets forth a flow chart illustrating anexemplary method for augmenting reality for a user according toembodiments of the present invention. The method of FIG. 3 includescreating (302) a model (124) of a region of the user's (140) environmentin the direction (133) of the user's field of vision (132). Theaugmented reality engine (122) of FIG. 1 may create (302) a model (124)of a region of the user's (140) environment in the direction (133) ofthe user's field of vision (132) by determining the direction (133) ofthe user's field of vision from the user's orientation and location, andmapping images of the user's environment to corresponding latitude,longitude, and altitude coordinates. As mentioned above, other data, inaddition to images of the user's environment, may be used to create amodel including for example, relief map data, blueprint data, and thelike.

The method of FIG. 3 also includes determining (304), in dependence uponthe model (124) of the region of the user's environment, that an objectlocated within the user's environment is invisible to the user (140).Determining (304) that an object located within the user's environmentis invisible to the user may be carried in various ways including, forexample, identifying the object's location (128) within the user'senvironment; identifying the user's location within the user'senvironment, and determining, from the model of the region of the user'senvironment, that there is an obstruction located between the user andthe object.

Another exemplary way of determining (304) that an object located withinthe user's environment is invisible to the user may include identifyingthe object's location (128) within the user's environment anddetermining, from the model of region of the user's environment, whetherthe object is located in an area of low light. Determining whether theobject is located in an area of low light may be carried out in numerousways including, for example, by determining from the time of day at thelocation of the object that the sun is set, determining from an infraredcamera that object is in a low light area and other ways as will occurto those of skill in the art.

Another exemplary way of determining (304) that an object located withinthe user's environment is invisible to the user may include identifyingthe object's location (128) within the user's environment anddetermining, from weather RADAR imagery, that the weather conditionssurrounding the object's location preclude visibility. Such weatherconditions may include, for example, dense fog, heavy rain, or sandstorms.

The method of FIG. 3 also includes determining (306), in dependence uponthe direction of the user's field of vision (132), the object's location(128), and the model (124) of the region of the user's environment, alocation (314) to display a representation of the object on atransparent display (102) in front of the user's (140) field of vision(132). Determining (306) a location to display a representation of theobject on a transparent display (102) may be carried out by identifyinga grid location of the object on the model of the region of the user'senvironment and mapping the grid location of the object on the model ofthe region of the user's environment to a grid location on thetransparent display. As mentioned above, the model (124) may beimplemented as a three dimensional map of the region of the user'senvironment defined by a grid representing latitude, longitude, andaltitude. A grid location of the object on the model of the region ofthe user's environment may include one or more three dimensionalcoordinate sets which may mapped to one or more two dimensionalcoordinates defining the grid on the transparent display. Typically arepresentation of the object will projected at a location on thetransparent display that corresponds to the location at which a userwould be capable of viewing the object if the object were visible to theuser.

The method of FIG. 3 also includes displaying (308) the representation(138) of the object at the determined location (314) on the transparentdisplay (102) in front of the user's field of vision (132). Theexemplary display module (120) of FIG. 3 may display (308) therepresentation (138) of the object on the transparent display (102) byprojecting the representation of the object on the transparent display.Displaying (308) the representation (138) of the object may also includedetermining a size of the display. A user may, for example, increase thesize of a representation of an object that is located a great distancefrom the user's location or decrease the size of a representation of anobject that is located near to the user's location.

The method of FIG. 3 also includes displaying (310), with therepresentation (138) of the object, contextual information (312)describing the object. Contextual information may be implemented invarious ways. Contextual information may be implemented, for example, astext describing various characteristics of the object. If the object isan automobile, for example, contextual information may be textdescribing the make, model, year, and vehicle identification number ofthe automobile. Contextual information may also be implemented, forexample, as symbols. If the object is a harmful gas or radiation, forexample, contextual information may be a warning symbol describing aharmful substance.

Exemplary embodiments of the present invention are described largely inthe context of a fully functional computer system for augmenting realityfor a user. Readers of skill in the art will recognize, however, thatthe present invention also may be embodied in a computer program productdisposed on signal bearing media for use with any suitable dataprocessing system. Such signal bearing media may be transmission mediaor recordable media for machine-readable information, including magneticmedia, optical media, or other suitable media. Examples of recordablemedia include magnetic disks in hard drives or diskettes, compact disksfor optical drives, magnetic tape, and others as will occur to those ofskill in the art. Examples of transmission media include telephonenetworks for voice communications and digital data communicationsnetworks such as, for example, Ethernets™ and networks that communicatewith the Internet Protocol and the World Wide Web as well as wirelesstransmission media such as, for example, networks implemented accordingto the IEEE 802.11 family of specifications. Persons skilled in the artwill immediately recognize that any computer system having suitableprogramming means will be capable of executing the steps of the methodof the invention as embodied in a program product. Persons skilled inthe art will recognize immediately that, although some of the exemplaryembodiments described in this specification are oriented to softwareinstalled and executing on computer hardware, nevertheless, alternativeembodiments implemented as firmware or as hardware are well within thescope of the present invention.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from its true spirit. The descriptions in thisspecification are for purposes of illustration only and are not to beconstrued in a limiting sense. The scope of the present invention islimited only by the language of the following claims.

1. A method of augmenting reality for a user, the method comprising:creating a model of a region of the user's environment in a direction ofthe user's field of vision; determining, in dependence upon the model ofthe region of the user's environment, that an object located within theuser's environment is invisible to the user; determining, in dependenceupon the direction of the user's field of vision, the object's location,and the model of the region of the user's environment, a location todisplay a representation of the object on a transparent display in frontof the user's field of vision; and displaying the representation of theobject at the determined location on the transparent display in front ofthe user's field of vision.
 2. The method of claim 1 further comprising:displaying, with the representation of the object, contextualinformation describing the object.
 3. The method of claim 1 whereindetermining, in dependence upon the model of the region of the user'senvironment, whether an object located within the user's environment isinvisible to the user; identifying the object's location within theuser's environment; identifying the user's location within the user'senvironment; and determining, from the model of the region of the user'senvironment, that there is an obstruction located between the user andthe object.
 4. The method of claim 1 wherein determining, in dependenceupon the model of the region of the user's environment, that an objectlocated within the user's environment is invisible to the user;identifying the object's location within the user's environment; anddetermining, from the model of user's environment, whether the object islocated in an area of low light.
 5. The method of claim 1 whereindetermining, in dependence upon the direction of the user's field ofvision, the location of the object, and the model of the region of theuser's environment, a location to display a representation of the objecton a transparent display in front of the user's field of vision furthercomprises: identifying a grid location of the object on the model of theregion of the user's environment; and mapping the grid location of theobject on the model of user's environment to a grid location on thetransparent display.
 6. The method of claim 1 wherein displaying therepresentation of the object at the determined location on thetransparent display in front of the user's field of vision furthercomprises projecting the representation of the object on the transparentdisplay.
 7. An apparatus for augmenting reality for a user, theapparatus comprising a computer processor, a computer memory operativelycoupled to the computer processor, the computer memory having disposedwithin it computer program instructions capable of: creating a model ofa region of the user's environment in a direction of the user's field ofvision; determining, in dependence upon the model of the region of theuser's environment, that an object located within the user's environmentis invisible to the user; determining, in dependence upon the directionof the user's field of vision, the object's location, and the model ofthe region of the user's environment, a location to display arepresentation of the object on a transparent display in front of theuser's field of vision; and displaying the representation of the objectat the determined location on the transparent display in front of theuser's field of vision.
 8. The apparatus of claim 7 further comprisingcomputer program instructions capable of: displaying, with therepresentation of the object, contextual information describing theobject.
 9. The apparatus of claim 7 wherein determining, in dependenceupon the model of the region of the user's environment, whether anobject located within the user's environment is invisible to the user;identifying the object's location within the user's environment;identifying the user's location within the user's environment; anddetermining, from the model of the region of the user's environment,that there is an obstruction located between the user and the object.10. The apparatus of claim 7 wherein determining, in dependence upon themodel of the region of the user's environment, that an object locatedwithin the user's environment is invisible to the user; identifying theobject's location within the user's environment; and determining, fromthe model of user's environment, whether the object is located in anarea of low light.
 11. The apparatus of claim 7 wherein determining, independence upon the direction of the user's field of vision, thelocation of the object, and the model of the region of the user'senvironment, a location to display a representation of the object on atransparent display in front of the user's field of vision furthercomprises: identifying a grid location of the object on the model of theregion of the user's environment; and mapping the grid location of theobject on the model of user's environment to a grid location on thetransparent display.
 12. The apparatus of claim 7 wherein displaying therepresentation of the object at the determined location on thetransparent display in front of the user's field of vision furthercomprises projecting the representation of the object on the transparentdisplay.
 13. A computer program product for augmenting reality for auser, the computer program product disposed in a computer readable,signal bearing medium, the computer program product comprising computerprogram instructions capable of: creating a model of a region of theuser's environment in a direction of the user's field of vision;determining, in dependence upon the model of the region of the user'senvironment, that an object located within the user's environment isinvisible to the user; determining, in dependence upon the direction ofthe user's field of vision, the object's location, and the model of theregion of the user's environment, a location to display a representationof the object on a transparent display in front of the user's field ofvision; and displaying the representation of the object at thedetermined location on the transparent display in front of the user'sfield of vision.
 14. The computer program product of claim 13 furthercomprising computer program instructions capable of: displaying, withthe representation of the object, contextual information describing theobject.
 15. The computer program product of claim 13 whereindetermining, in dependence upon the model of the region of the user'senvironment, whether an object located within the user's environment isinvisible to the user; identifying the object's location within theuser's environment; identifying the user's location within the user'senvironment; and determining, from the model of the region of the user'senvironment, that there is an obstruction located between the user andthe object.
 16. The computer program product of claim 13 whereindetermining, in dependence upon the model of the region of the user'senvironment, that an object located within the user's environment isinvisible to the user; identifying the object's location within theuser's environment; and determining, from the model of user'senvironment, whether the object is located in an area of low light. 17.The computer program product of claim 13 wherein determining, independence upon the direction of the user's field of vision, thelocation of the object, and the model of the region of the user'senvironment, a location to display a representation of the object on atransparent display in front of the user's field of vision furthercomprises: identifying a grid location of the object on the model of theregion of the user's environment; and mapping the grid location of theobject on the model of user's environment to a grid location on thetransparent display.
 18. The computer program product of claim 13wherein displaying the representation of the object at the determinedlocation on the transparent display in front of the user's field ofvision further comprises projecting the representation of the object onthe transparent display.
 19. The computer program product of claim 13wherein the signal bearing medium comprises a recordable medium.
 20. Thecomputer program product of claim 13 wherein the signal bearing mediumcomprises a transmission medium.